Multi-context conversational environment system and method

ABSTRACT

An interactive speech-activated information retrieval application for use in automated telephone systems includes a control manager that interfaces between the caller&#39;s speech input and applications and enables several applications to be open at the same time. The control manager continually monitors for control words, enabling the user to switch between applications at will. When a user switches to another application, the control manager suspends the first application and stores its context, enabling the user to later return to the application at the point where the application was previously suspended.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application and claims priority toU.S. patent application Ser. No. 09/870,202, filed May 30, 2001,entitled “Multi-Context Conversational Environment System And Method”,which is hereby incorporated by reference in its entirety.

FIELD OF THE INVENTION

The present invention relates in general to speech recognition and, moreparticularly, to interactive speech applications for use in automatedtelephone assistance systems.

BACKGROUND OF THE INVENTION

Computer-based interactive speech applications are widely used intelephone systems to answer incoming calls as well as to perform varioustasks including gathering information from callers, providinginformation to callers, and connecting callers with appropriate parties.Typically, the call flow is activated when the system receives anincoming call, and begins with a system-generated greeting followed by alist of available options. The application waits for a response from thecaller and processes the response when received. The application must beable to recognize what the caller said to determine what should happennext. It does this by converting the user's speech into a text sentenceof distinct words (speech recognition), breaking down the recognizedsentence grammatically and then systematically representing its meaning(language understanding), obtaining targeted data based on thatmeaning's representation from an appropriate online source (informationretrieval), building a text sentence that presents the retrieved data inthe user's preferred language (language generation), and converting thattext sentence into computer-generated speech (speech synthesis).

Typically, interactive speech applications today are “single-threaded.”If a user accesses an interactive speech-activated system to receiveinformation, he would select one of the available options, complete theprocess and then return to the main menu to select a second option, ifdesired. If a user abandons the process before completion, thenavigation through the system so far is lost and the user must startagain at the top of the call flow. It would be helpful if a user wereable to select an option, go partway through the call flow, decide toaccess another thread before proceeding with the first thread, suspendthe first thread, select a second thread, and thereafter be able toreturn to the first thread at the point he left it. Hence a need existsfor a multi-contextual speech application that overcomes the drawbacksof the prior art.

SUMMARY OF THE INVENTION

The present invention is directed to providing a multi-contextual speechapplication environment wherein several applications can be activated atthe same time, thereby enabling the user to switch between applicationsat will while maintaining each application context. A control managerstarts up applications, monitors speech for specific “control words” or“control phrases”, and switches control among applications. Eachapplication is independent of other applications, interacting with thecontrol manager but not directly interacting with other applications,thereby leading to ease of development and maintenance because the firstapplication does not need to be aware of the existence of other activeapplications.

The foregoing and other aspects of the present invention will becomeapparent from the following detailed description of the invention whenconsidered in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a block diagram of a general speech recognition system;

FIG. 2 shows a more detailed block diagram of the system of FIG. 1;

FIGS. 3 and 4 depict portions of an exemplary call flow as is well knownin the art;

FIG. 5 shows a block diagram of an exemplary speech recognition systemin accordance with the present invention;

FIG. 6 shows a more detailed block diagram of an exemplary speechrecognition system in accordance with the present invention; and

FIG. 7 shows a flow diagram of an exemplary method of speech recognitionin accordance with the present invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS AND BEST MODE

Overview

Speech recognition systems have been developed in many parts of theworld and, although it is difficult to describe a standard recognitionsystem architecture, some characteristics are shared between many ofthem. A typical speech recognition system, of the type depicted in FIG.1, generally comprises a device such as a microphone or telephone set101 that converts a spoken utterance into an electric signal andtransmits the signal to a speech recognition unit 100. The speechrecognition unit 100 can be split into two functional blocks: aprocessing unit 102 and a search unit 104. The processing unit 102 is anacoustic processor that performs the segmentation, the normalization,and the parameterization of the input signal waveform. In some cases,especially for connected word speech, this stage may also include afeature extraction operation. The search unit 104 includes a speechrecognition dictionary that is scored in order to find possible matchesto the spoken utterance.

More specifically, the processing unit 102, illustrated in greaterdetail in FIG. 2, translates the incoming analog speech waveform intodigital format. This can be done with the use of an A/D converter 110, aspectrogram generator or any other suitable device or technique. Theinput signal is then split into short segments called analysis frameswhose typical duration ranges from about 5-20 ms. Further processingwill be done relative to these frames. The processing unit 102 furthercomprises a feature extractor 112, which can comprise a normalizer and aparameterizer, and an endpoint detector 114.

The normalizer adjusts the maximum signal amplitude of each analysisframe to a standard level in order to take into account variations inspeech intensity, transmission losses and other physical effects such asdistance from the microphone and recording level. The parameterizertypically represents speech frames in terms of voicing decision,amplitude and fundamental frequency. A wide variety of parameters can beused in the parameterizer.

The endpoint detector 114 splits the input signal waveform into astarting point and an endpoint of the speech utterance. This stage usesalgorithms whose purpose is to locate the boundaries between silence andspeech. Many systems use the short term energy and the zero crossingrate as an indication of the beginning or end of a word. Moreover,typical endpoint detection units use many parameters including frameenergy, frame voice labels and other statistical variance parametersderived from speech.

The search unit 104, shown in more detail in FIG. 2, scores or otherwiseranks all the words (also known as “orthographies”) in a speechrecognition dictionary database 120 such as to be able to derive theorthography or orthographies which have the highest probability ofmatching the spoken utterance. A comparator and scorer 122 compares thespoken utterance with the entries in the database 120 to determine amatch based on closest score. Typical algorithms that can be usedinclude the fast score estimation and the graph search algorithms, asknown by those skilled in the art.

The accept/reject stage 124 compares the score to a predeterminedthreshold to determine if a correct mapping was found or if no mappingcould be determined. If the score is greater than or equal to thepredetermined threshold, then it is determined that a correct mappinghas been found and the recognized word is provided as output, generallyto another part of the system which may use the output as a link toinformation. If the score is below the threshold, no match is found andthe input spoken utterance is rejected.

FIGS. 3 and 4 depict the call flow of an illustrative interactive speechapplication 300 as is presently known in the art, for use by a caller toobtain information. Application 300 may be executed by a voiceprocessing unit in a telephone system. The call flow is activated whenthe system receives an incoming call, and begins by outputting agreeting at step 310, such as, “Welcome to Info by Voice for Atlanta.Please tell me the information category you want now or say ‘Help’ and Iwill explain your choices.” In this example, the application outputs anaudible speech signal to the caller by, for example, playing apre-recorded prompt or using a speech generator such as text-to-speechconverter to produce the message.

If the user responds, “Help” at step 315, the application then listsavailable options to the caller at step 320, by outputting a messagesuch as, “Please listen carefully to the following eleven choices.Remember you can speak your request at any time: Restaurant Guides,Ticket Information, Stock Quotes, Weather Forecast, Business News,Sports, Daily Horoscopes, Lottery Results”.

If the caller does not say “Help” at step 315 or after the options arelisted at step 320, the system waits for a response from the caller atstep 330. At step 340 the application determines one or more potentialmatches between the user's input and the available options, creating aset of hypotheses. If the caller says, for example, “Restaurants”, theapplication must be able to recognize what the caller said and determinewhether the caller's spoken input corresponds to one of the choiceslisted in the options. The application may determine that there are aplurality of potential matches between the caller's spoken input and thelist of possible choices. At step 342, if there are a plurality ofpotential matches, the application asks the user to repeat his selectionand processing returns to step 330, to wait for the caller's response.

If the set of hypotheses contains only one potential match but the scoreof the recognized string is lower than a predetermined confirmationthreshold, at step 350 the application attempts to confirm the caller'sresponse. Typically, the application will attempt to verify what thecaller said, by asking, for example, “I think you said ‘Restaurant’. Isthat correct?”

At step 360, the application once again waits to receive a response fromthe caller. If the caller says “yes” and his response is understood bythe application, processing continues at step 370. If the caller says“no” the caller will be re-prompted for input (not shown). When thecaller's confirmation, (“Yes”), the application takes the appropriatesteps to initiate the correct process at step 370, which in the givenexample is process 305 a. This process repeats for each acceptablerecognition result.

If the set of hypotheses contains only one potential match and the scoreof the recognized string is higher than the predetermined confirmationthreshold, processing continues at step 370.

Referring now to FIG. 4, the application then prompts the caller for aresponse by outputting a message such as “Welcome to ‘Restaurant’. Doyou want to search by restaurant name or by type of food?” at step 325a. The system then waits for a response from the caller, at step 330 a,and processes the response when received at step 340 a. If the callersays, for example, “Type of food” the application must be able torecognize what the caller said and determine what information toprovide. The application again attempts to verify what the caller said,at step 350 a, by asking another question requiring a response, such as“I think you said ‘Search by type of food’. Is that correct?” If thecaller responds, at step 360 a, “Yes”, the speech application will thensay, for example, “Tell me the type of food you want. You may alsoinclude the area of town or landmark you want.” After this informationis received from the caller (e.g., “Chinese restaurants in Midtown”) atstep 370 a, the application lists the Chinese restaurants in Midtown atstep 380 a.

These types of conversational environments offered by current speechrecognition applications allow a caller to interact using only aone-to-one dialog mode. For example, in order to obtain informationabout another topic, such as “Weather” in the example above and shown inprocess 305 b, the caller must exit the “Restaurant” application andreturn to the main menu by saying “Main Menu”, or go to the beginning ofthe weather service by saying “Weather”. Processing then continuessimilar to that described above with respect to process 305 a, but witha different set of prompts and information.

Additionally, whenever a request to switch to another topic option isreceived, processing for the first application terminates, and the callthread is lost. That is, if a caller who has navigated to “Chineserestaurants in Midtown” switches to “Weather” and then returns to“Restaurants”, the caller will have to start again at the top, “Do youwant to search by name of restaurant or by type of food?” rather thanpicking up at listing the Chinese restaurants in Midtown.

Multi-Context Conversational Environment

The present invention is directed to systems and methods for providing amulti-contextual speech application environment wherein severalapplications can be open at the same time, enabling a caller to switchbetween applications at will. A control manager interfaces with thecaller, starts up applications, switches control among applications,alternately suspends and re-activates open applications, and maintainsthe context of suspended applications. Each application may beindependent of other applications. Alternately the context may betransferred across applications. For example if a user who asked for“Chinese in Atlanta”, then switched to weather, the system may ask theuser if the user in interested in Atlanta weather. Additionally, thechance of recognizing “Atlanta” compared to other cities may beincreased.

The present invention enables a caller to open multiple threads at thesame time. For example, the present invention would enable a caller tonavigate to “Chinese restaurants in Midtown,” and then switch to“Weather,” and then return to “Restaurants” and once again be at thepoint where the Chinese restaurants in Midtown would be listed.Moreover, any number of applications can be open at any time, limited bystorage and other processing considerations. Such an environment helpssimplify the application design because the application designerconcentrates only on the designer's application. System users, however,may mix and match the functionalities of multiple applications accordingto the user's needs.

FIG. 5 illustrates an exemplary environment in which the presentinvention may be deployed. As can be seen, a caller connects to acommunications network 504 such as the publicly switched telephonenetwork (PSTN) though a communications device 502 such as a telephone toreceive audible information from a database of information stored on andprocessed by a server 506. Components of a typical server computer mayinclude, but are not limited to, a processing unit, a system memory, asystem bus that couples various system components including the systemmemory to the processing unit, storage media and various input andoutput devices.

The communications device 502 may include a subscriber telephone setthat is connected via a telephone line (e.g., POTS, or similar) to atelephone system (e.g., communications network 504) including at leastone central office switch, at least one service control point (SCP), anda service node (SN). This exemplary environment is a PSTN. It iscontemplated that a portion of the PSTN may be part of the AdvancedIntelligent Network (AIN) of a typical local exchange carrier. Forfurther information regarding the referenced PSTN and AIN aspectsthereof, the interested reader is referred to the patent to Weisser,U.S. Pat. No. 5,430,719, which is incorporated herein by reference. TheSN typically can include voice and dual tone multi-frequency (DTMF)signal recognition devices and voice synthesis devices which can be usedin accordance with the present invention.

FIG. 6 shows a more detailed block diagram of an exemplary speechrecognition system in accordance with the present invention. Theconversational operating environment 600 acts as a mediating agentbetween a caller's speech input 630 and the application software 620 a,620 b, 620 c through speech input and voice prompting output. Each openapplication 620 a, 620 b, 620 c is assigned to a task agent, analogousto a window in the graphical world. Control manager 604 starts upapplications, assigns a newly started application 620 a, 620 b, 620 c toa task agent 610 a, 610 b, 610 c, switches control among task agents 610a, 610 b, 610 c, maintains the context for each application in contexttable 602, and may transfers context across applications. The caller cancall up control manager 604 at any time from any application 620 a, 620b, 620 c by speaking one of a list of control words or phrases. Thecaller interacts with the operating environment 600 and at any giventime operating environment 600 directs the caller's speech input 630,when appropriate, to the currently active task agent 610 a, 610 b, 610c.

Because almost all of the processing for current over-the-telephonespeech applications occur on the server side (the client is merely atelephone and has little or no processing power), the currentenvironment states for a particular caller can persistently be saved andrestored across sessions in context table 602 for each caller. Accordingto one aspect of the invention, a caller may request to be reminded ofthe last context for an application 620 a, 620 b, 620 c. Control manager604 will output to the caller the context stored in context table 602for the application 620 a, 620 b, 620 c. Although three applications aredescribed with respect to FIG. 6, it is contemplated that any number ofapplications can be implemented.

FIG. 7 shows a flow diagram of an exemplary method of speech recognitionin accordance with the present invention. At step 704, a callerinitiates a call to an information service in accordance with thepresent invention. At step 708, a welcome message is desirably played tothe caller and the caller thereafter is prompted for the name of adesired service application (for example, a subject area such as“Weather”, “Stock Quotes”, etc.). At step 712, the caller indicates thedesired application or subject area by speaking into the telephonehandset, for example. The speech input is then processed/searched todetermine the desired application. If the speech input does notcorrespond to an application, the user is prompted to repeat the speechinput, for example, or the call may be forwarded to a human operator forassistance.

At step 716, the control manager initiates a task agent for theappropriate (user selected) application, as described above. The taskagent then activates the application, at step 720. At step 724, theapplication prompts the caller for input through the control manager.The input is used by the application to narrow or limit the search(e.g., after choosing “Restaurants”, the input may be for the type ofrestaurant or location, for example). At step 728, the input is receivedby the control manager and processed/searched to determine whether theinput corresponds to available selections. If the speech input does notcorrespond to an application, the user is prompted to repeat the speechinput, for example, or the call may be forwarded to a human operator forassistance.

The control manager monitors the input received and analyzes the inputfor the presence of command words, such as those that correspond tostarting another application or subject area (e.g., “Main Menu”,“Lottery Results”, “Stock Quotes”, etc.). At step 732, if no commandwords are found, the input is passed to the current speech applicationat step 736, which uses it in the current search or thread.

On the other hand, if at step 732 it is determined that the caller'sinput does contain a command word or words, the control manager savesthe context of the currently open application in the context table atstep 740. At step 744, the command word is compared to a list of wordsindicating that the caller wishes to terminate the call (e.g.,“Goodbye”, “End”, etc.). If it is determined that the caller wishes toterminate the call, the system exits at step 756. If termination is notindicated, the command word is compared to a list of possibleapplications to determine the next application to be initiated, at step748. If a valid service is found, the control manager suspends the firstapplication at step 752 and initiates a task agent for the secondapplication, with processing continuing at step 716. At least oneindicator is stored that indicates the current processing step of theinitial application. In this manner, the caller can return to theapplication at the point where it was suspended when the secondapplication was opened. Preferably, the at least one indicator comprisesa series of indicators that indicate a processing path of the firstapplication. Moreover, desirably, a computer-generated representation ofthe series of indicators is stored (for future use or analysis) thatindicates the processing path of the first application.

As an example, assume that a caller initiates a call to a telephoneinformation service in accordance with the present invention in whichthe information provided includes weather, airline reservations andhotel reservations. The caller may indicate “Airline reservation” whenprompted for his choice of application, and navigate to the point wherehe will receive information concerning the cost of a flight from Atlantato Philadelphia on August 4. At this point the caller may say “Hotel” toindicate that he wishes to receive information concerning hotelreservations. After determining the cost or availability of hotelreservations in Philadelphia, the caller could indicate his desire toreturn to the “Airline Reservation” application by saying “Airline” andwould be returned to the point at which he suspended the “AirlineReservation” application to inquire about hotel reservations. Althoughin this example only two contexts are illustrated, it should beunderstood that any number of threads or applications could be active atthe same time, limited only by the capacity of the context table.

It is contemplated that each application can provide the audibleinformation to the caller in an associated voice, so that differentvoices are used for different applications. In this manner, a callerwill hear distinguishable voices providing information for the variousapplications.

Although illustrated and described herein with reference to certainspecific embodiments, the present invention is nevertheless not intendedto be limited to the details shown. Rather, various modifications may bemade in the details within the scope and range of equivalents of theclaims and without departing from the invention.

1. A method of speech recognition processing that provides audibleinformation over a communications device comprising: receiving a firstspeech input indicative of a first subject area; initiating a firstsubject application associated with said first subject area; receiving asecond speech input indicative of a second subject area; and storing atleast one indicator indicating a current processing step of said firstsubject application.
 2. The method according to claim 1, furthercomprising initiating a second subject application associated with saidsecond subject area.
 3. The method according to claim 1, furthercomprising initiating a task agent for said first subject application.4. The method according to claim 1, further comprising monitoring saidfirst speech input for at least one word indicative of said secondsubject area.
 5. The method according to claim 1, further comprisingsuspending said first subject application after receiving said secondspeech input.
 6. The method according to claim 5, further comprising:receiving a further speech input, and re-activating said first subjectapplication responsive to the further speech input.
 7. The methodaccording to claim 1, wherein said storing at least one indicatorfurther comprises storing a series of indicators that indicate aprocessing path of said first application.
 8. The method according toclaim 7, further comprising outputting a computer-generatedrepresentation of said stored series of indicators that indicates saidprocessing path of said first application.
 9. The method according toclaim 1, further comprising outputting information associated with saidfirst application in a first voice.
 10. The method according to claim 9,further comprising outputting information associated with said secondapplication in a second voice, said second voice being distinguishablefrom said first voice.
 11. The method according to claim 1, furthercomprising synthesizing speech for outputting speech from said firstapplication.
 12. A speech recognition system comprising: a speechrecognition module that processes speech input and translates saidspeech input into computer-readable input; a control manager comprising:a module that interfaces between said speech input and at least one of aplurality of application programs; a module that initiates processing ofa first application program; and a module that monitors said speechinput for a request to initiate a second application program; and aspeech synthesizing module for providing output information from saidplurality of application programs.
 13. The system according to claim 12,further comprising a context table for maintaining a context for each ofsaid plurality of application programs.
 14. The system according toclaim 12, further comprising a plurality of task agents, each task agentassociated with one of said plurality of application programs.
 15. Thesystem according to claim 14, wherein the control manager is adapted toassign said application programs to said task agents, and switch controlamong said task agents.
 16. The system according to claim 12, whereinthe request comprises a control word.
 17. The system according to claim12, wherein the control manager is adapted to suspend said firstapplication program, and initiate processing of said second applicationprogram, responsive to the request.
 18. The system according to claim17, wherein the control manager is adapted to re-activate said firstapplication program responsive to a further request.
 19. The systemaccording to claim 12, wherein the control manager is adapted to storeat least one indicator indicative of a current processing step of atleast one of said plurality of application programs.
 20. Acomputer-readable medium for storing computer-executable instructionsfor performing the method of claim 1.